Incremental Without Replacement Sampling in Nonconvex Optimization

Minibatch decomposition methods for empirical risk minimization are commonly analyzed in a stochastic approximation setting, also known as sampling with replacement. On the other hand, modern implementations of such techniques incremental: they rely on without replacement, which available analysis is much scarcer. We provide convergence guaranties latter variant by analyzing versatile incremental gradient scheme. For this scheme, we consider constant, decreasing or adaptive step sizes. In smooth obtain explicit complexity estimates terms epoch counter. nonsmooth prove that sequence attracted solutions optimality conditions problem.